Sparse Recovery in Convex Hulls of Infinite Dictionaries

Let S be an arbitrary measurable space, T ⊂ R and (X,Y ) be a random couple in S × T with unknown distribution P. Let (X1, Y1), . . . , (Xn, Yn) be i.i.d. copies of (X,Y ). Denote by Pn the empirical distribution based on the sample (Xi, Yi), i = 1, . . . , n. Let H be a set of uniformly bounded functions on S. Suppose that H is equipped with a σ-algebra and with a finite measure μ. Let D be a convex set of probability densities with respect to μ. For λ ∈ D, define the mixture fλ(·) = ∫ H h(·)λ(h)dμ(h). Given a loss function ` : T × R 7→ R such that, for all y ∈ T, `(y, ·) is convex, denote (` • f)(x, y) = `(y; f(x)). We study the following penalized empirical risk minimization problem λ̂ε := argmin λ∈D [ Pn(` • fλ) + ε ∫ λ log λdμ ] along with its distribution dependent version λε := argmin λ∈D [ P (` • fλ) + ε ∫ λ log λdμ ] . We prove that the “approximate sparsity” of λε implies the “approximate sparsity” of λ̂ε and study connections between the sparsity and the excess risk of empirical solutions λ̂ε.